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A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
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DETAILED ACTION 

y 

Response to Arguments 

Applicant argues that Ahmad does not teach one to form a model of a multimedia 
presentation based upon event cues, and then classifying that presentation based upon 
how well it matches such a model, as recited in claim 1 element (a). 

The Examiner respectfully disagrees. Ahmed meets the broadly claimed "event 
cues" as discussed in the previous office action. Since the "event cues" are not further 
defined, Ahmed meets the claimed limitation. 

Applicant argues the distinction between keywords and named entries and that 
Ahmed uses keywords and not name entries. 

The Examiner respectfully disagrees. Ahmed discloses the broadly claimed 
"named entities" and in those claims where named entities are more clearly defined it 
would have been obvious to modify Ahmed to include the claimed limitation. As a 
result, the Applicant's arguments are not persuasive. 

Applicant argues that Ahmed's "update from" is not an introductory term. The 
Examiner respectfully disagrees. The word "update from" meets the broadly claimed 
introductory term. As a result, the Applicant's argument is not persuasive. 

Applicant argues that Ahmed fails to disclose the claimed story segments. The 
Examiner respectfully disagrees. The "headline news" in fig 2B meets the broadly 
claimed limitation. 
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Claim Rejections - 35 U.S.C. § 102 

I. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 
that form the basis for the rejections under this section made in this Office 
action: 

A person shall be entitled to a patent unless - 

(e) the invention was described in a patent granted on an application for patent by another filed in 
the United States before the invention thereof by the applicant for patent, or on an international 
application by another who has fulfilled the requirements of paragraphs (1), (2), and (4) of section 
371© of this title before the invention thereof by the applicant for patent. 

The changes made to 35 U.S.C. 102(e) by the American Inventors Protection Act 

of 1999 (AIPA) do not apply to the examination of this application as the application 

being examined was not (1 ) filed on or after November 29, 2000, or (2) voluntarily 

published under 35 U.S.C. 122(b). Therefore, this application is examined under 35 

U.S.C. 102(e) prior to the amendment by the AIPA (pre-AlPA 35 U.S.C. 102(e)). 

II. Claims 1-11, 13, 15, 17, 19, 20-24, and 28-54 are rejected under 35 U.S.C. 
102(e) as being anticipated by Ahmed. 

Considering claims 1 , 40, 43, 44, and 55 Ahmed discloses a system for 
automatically processing a plurality of streams (multi info streams are met by 'audio', 
'video 1 and 'text' see col 33 lines 49-52) to classify event cues within the received data 
stream, wherein the event cues are text data, caption data, audio data and video data 
(see col 15 lines 25-42, col 17 lines 26-40, col 23 lines 10-45 and col 24 lines 18-63). 
The event cues detected are compared or matched to existing cues in specific classes 
or categories to classify the incoming data into an appropriate category or class (see col 
17 lines 25-41 and col 18 lines 25-40). By searching through the classified cues, 
Ahmed discloses comparing and matching the unclassified cues with the classified cues 
to correctly classify and place the story segments in the correct category or 



Application/Control Number: 09/033,268 
Art Unit: 2611 



Page 4 



classification. Thus Ahmed discloses selecting at least one contiguous portion of the 
multimedia presentation as a story segment, by matching a model of expected event 
cues with observed event cues in at least one information stream and to classify the 
presentations to which specific multimedia presentation belongs. The claimed story 
segment is depicted by Ahmed in fig 2-b. In particular figure 2-b depicts classifying 
news into a plurality of categories (see items 215 and 214) wherein a story segments 
are provided for each category (see "Erin: A Tropical Storm, etc.). Further Ahmed 
discloses extracting text data from a text information stream which corresponds to the 
story segment (see col 23 lines 35-59 and col 17 lines 25-41 ), extracting named entities 
from the text data (see col 23 lines 55-59 and col 15 lines 25-41 ) and extracting story 
summary data using the named entities as a basis (see col 15 lines 25 - 40 and fig 2-b, 
the news and sports stories retrieved are classified under "news" or "sports"). Further, 
Ahmed discloses displaying "keyframes" or "thumbnail" images (see col 16 lines 64-67) 
which is inherently extracted from the "imagery system" since it is displayed (see fig 2-b 
item 213 "Headline News"). Ahmed clearly depicts displaying the keyframe "Headline 
News" with the story summary data (see fig 2-b). Ahmed also discloses selecting a 
keyframe enabling a user to display a representative segment. In particular, by 
selecting keyframes "Headline News" or "Sports" in fig 2-b a user can access a 
representative story segment(s). 

Considering claims 2 and 30, Ahmed inherently discloses a "machine" (met by 
system controller 103, data storage device and primary and secondary acquisition 
device in fig 1) which compare detected cues including transitions in data (met by "we 
go now to" or "update from" in col 24 lines 37-41) to existing or expected developed 
from observed event cues occurring in a class of media presentations cues to classify 
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the detected cues and representative story segments in the correct category or class 
(see col 17 lines 25-41 and col 18 lines 25-40). 

Considering claim 3, Ahmed discloses the cues can be audio clues by detecting 
pause recognition in audio (see col 24 lines 58-60). Ahmed discloses the claimed 
"plurality of time difference based cues" because a large time difference in words or 
sounds in the audio would indicated a pause in the audio. 

Considering claim 4, Ahmed discloses the claimed "wherein the expected event 
cues comprise a plurality of intrastream cues taken from a given information stream" 
(see col 23 lines 55-60 and col 24 lines 39-40, the cues "we go now to" and "update 
from" are taken cues take within the middle of the information stream or "intrastream"). 

Considering claims 6-11, Ahmed discloses detecting specific cues used in 
classifying data including text information, closed caption words, closed caption 
punctuation cues, token phrases, text strings (col 17 lines 24-41, col 23 lines 18-22, col 
23 lines 54-59, col 24 lines 37-41). 

Considering claim 13, Ahmed discloses the claimed "named entity" and "text 
string" (see col 23 lines 56-59). 

Considering claim 15, Ahmed discloses the claimed "named entity" and "closed 
caption punctuation clue" (see col 23 lines 55-59, note: punctuation met by comma's). 

Considering claim 17, Ahmed discloses the claimed "introductory news broadcast 
terms" (see col 24 lines 37-41 , met by "update from"). 

Considering claims 19 and 22, Ahmed discloses the claimed "anchor to reporter 
hand-off phases (see col 23 lines 39-41). 

Considering claims 20, 21 and 23, Ahmed discloses the claimed "station" and 
"reporter named entity" (see col 23 lines 25-60). 



Application/Control Number: 09/033,268 



Art Unit: 2611 



Page 6 



Considering claim 24, Ahmed discloses the claimed "leaders to highlights of 
upcoming news stores" (see col 24 lines 37-41 , met by "more on that" which is a leader 
to an upcoming story). 

Considering claims 28 and 29, Ahmed discloses the claimed "named entities" 
(see col 23 lines 55-59, see "Jane Doe" for name or "Anytown, USA" for location). 

Considering claim 31, Ahmed discloses the claimed "wherein the model of 
expected even cues is developed by statistical analysis of observed event cues" (see 
col 25 lines 30-67, col 28 lines 35-55, col 29 lines 45-65). 

Considering claim 32, Ahmed discloses taking cues from a video stream which 
inherently comprises images and thus discloses the claimed "expected event cues are 
taken from an image information stream" (see col 25 lines 30-40). 

Considering claim 33, Ahmed discloses the claimed "reporter frame" (see col 23 
lines 55-59 and col 24 lines 37-41). 

Considering claim 34, Ahmed discloses the claimed "wherein the event cues are 
taken from an audio information stream" (see col 24 lines 42-46). 

Considering claim 35, Ahmed discloses a cues from audio pause recognition 
which meets the claimed "silence detection" (see col 24 lines 58-60). 

Considering claim 36, Ahmed discloses the claimed "news stories" (see col 15 
lines 25 - 42). 

Considering claim 37, Ahmed discloses cues which start a story segment (see 
col 24 lines 39-41 see "we go now to"). 

Considering claims 38 and 47, Ahmed discloses a method of processing 
representation of multimedia presentation having multi info streams (multi info streams 
are met by 'audio', Video' and 'text' see col 33 lines 49-52) comprising the steps of 
summarizing segments of text, audio and video see col 33 lines 33-61) and thus 
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discloses the claimed "selecting at least one contiguous portion of the multimedia 
presentation as a story segment". Further, since text data is extracted from the 
received stream and is used to compile the summary, Ahmed discloses "extracting text 
information from a text information stream corresponding to the story segment as text 
data" and "extracting story summary data from the text data (see col 33 line 49 - col 34 
line 39). Ahmed further discloses linking together a stored representation of the text 
data, summary data and name entity data (see fig b, "Erin: A Tropical Storm" meets the 
claimed 'named entity data 1 and 'text' and 'summary 1 data are met by the data 
description of "Erin: A Tropical Storm"). Ahmed also discloses displaying multiple 
summary elements representative of the extracted story segments (see fig 2-b which 
displays multiple stories wherein each story has multiple elements including the title, 
description, time and location). 

Considering claim 39, Ahmed discloses storing the all the data including 
summary data, named entity data for story segments ans one or more files one a file 
server (file server met by "data storage device 104" in fig 1 ). Further, the server is 
connected to the Internet which meets the "computer network" limitation (see col 1 1 
lines 40-65). Further, since a user can "browse" the data and categories, Ahmed 
discloses the claimed "allowing access to the stored representations of news story 
segments available to a browser program running on the client computer connected to 
the network 1 (see col 1 1 lines 31-65). 

Considering claim 41, Ahmed discloses detecting the number of words occurring 
in a segment as a means for selecting a segment as a category or class (see col 28 
lines 36-55) and thus discloses the claimed "determining a frequency of occurrence for 
the named entities per sentence in a story segment and selecting a sentence with a 
greatest named entity frequency of occurrence as a topic sentence". 
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Considering claim 42, Ahmed discloses detecting which segment has a greater 
similarity to a primary segment (see col 29 lines 18-44) including detecting the 
frequency or number of words or "greatest named entity frequency of occurrence" in a 
segment (see col 28 lines 26-55) and thus discloses the claimed "if more than one 
sentence has a greatest named entity frequency of occurrence, selecting the sentence 
closest to the beginning of the story as the topic sentence" limitation. 

Considering claim 45, Ahmed discloses wherein the claimed "wherein the story 
segment contains a reporter segment and the key frame is selected from the middle of 
the reporter segment (see col 24 lines 37-41 , the phrase "we go now to" or "more on 
that" is the middle of a reporter segment). 

Considering claim 46, Ahmed discloses the claimed "wherein the story segment 
contains an anchor booth segment and the keyframe is selected from the middle of the 
anchor booth segment" (see col 24 lines 37-41 , phrases "we go now to", "update from" 
or "more on that" meets the claimed middle of the anchor booth segment). 

Considering claim 48, Ahmed discloses the plurality of elements of the story 
displayed together including the title, description, time and location (see fig 2-b). 

Considering claim 49, Ahmed discloses the name of the summary (see "Erin: A 
Tropical Storm" in fig 2-b) and thus discloses the claimed limitation. 

Considering claim 50, Ahmed discloses the claimed "summary of sentences" 
(see description of "Erin: A Tropical Storm" in fig 2-b). 

Considering claim 51, Ahmed discloses the claimed summary presentation 
elements comprise a hyperlink to a source media element of the segment (see fig 2-b, 
hyperlinks 213 and 214 take user to source of information). 

Considering claim 52, Ahmed discloses the claimed hyperlinks which lead to 
presentation of additional elements of the story segment including named entities (see 
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fig 2-b, selecting "Headline News" leads to headline news stories and selecting the 
sports link 215 leads to sports news, stories and named entities). 

Considering claims 53 and 54, Ahmed discloses selecting a hyperlink for 
searching for related stories and presenting a thumbnail view comprising key frames 
from the multiple story segments (see fig 2-b, by selecting "Headline News", news under 
"Headline News" is searched and retrieved and displayed including the "Headline News" 
keyframe 213 and multiple story segments 214). 

Claim Rejections - 35 U.S.C. § 103 

III. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for 

all obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as 
set forth in section 102 of this title, if the differences between the subject matter sought to be 
patented and the prior art are such that the subject matter as a whole would have been obvious 
at the time the invention was made to a person having ordinary skill in the art to which said 
subject matter pertains. Patentability shall not be negatived by the manner in which the invention 
was made. 
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IV. Claims 12, 14, 16, 18, and 25-27 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Ahmed. 

Considering claim 12, Ahmed discloses cues including a name followed by a 
comma's in a sentence as a means for indicating a break between segments. It would 
have been obvious modifying Ahmed to include ":" would have had the same effect of 
quickly identifying a break in a segment. Therefore, it would have been obvious to one 
having ordinary skill in the art at the time the invention was made to modify Ahmed to 
include the claimed ":" to quickly identify a break in a segment. 

Considering claim 14, Ahmed discloses cues including a token phrase "Jane 
Doe, WXYZ news, reporting live from Anytown, USA" to indicate a break between 
segments. It would have been obvious modifying Ahmed to include Tm" followed by 
the person's name would have also provided a cue indicating a break between 
segments since news reporters often identify themselves by Tm" followed by their 
name. Therefore, it would have been obvious to one having ordinary skill in the art at 
the time the invention was made to modify Ahmed to include the claimed "I'm" followed 
by a name to quickly indicate a break between segments since this phase is often used 
by reporters. 

Considering claim 16, Ahmed discloses cues including a name followed by a 
comma's in a sentence as a means for indicating a break between segments. It would 
have been obvious modifying Ahmed to include a name followed by ":" would have had 
the same effect of quickly identifying a break in a segment. Therefore, it would have 
been obvious to one having ordinary skill in the art at the time the invention was made 
to modify Ahmed to include the claimed name followed by ":" to quickly identify a break 
in a segment. 
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Considering claim 18, Ahmed discloses cues including a token phrase "Jane 
Doe, WXYZ news, reporting live from Anytown, USA" to indicate a break between 
segments. It would have been obvious modifying Ahmed to include Tm" would have 
also provided a cue indicating a break between segments since news reporters often 
identify themselves by "Tm" followed by their name. Therefore, it would have been 
obvious to one having ordinary skill in the art at the time the invention was made to 
modify Ahmed to include the claimed "Tm" to quickly indicate a break between 
segments since this phase is often used by reporters. 

Considering claim 25, Ahmed discloses using token phrases like "we go now to" 
and "more on that". It would have been obvious token phrases like "coming up" and 
"next on" would have also indicated a break between segments since these phrases are 
often used in the news. Therefore, it would have been obvious to one having ordinary 
skill in the art at the time the invention was made to modify Ahmed to include the 
claimed phrases to quick identify breaks or transitions in the segment since these 
phrases are often used. 

Considering claim 26, Ahmed discloses the claimed sign off phrase (see col 23 
lines 55-59 "Jane Doe, WXYZ news, reporting live from Anytown, USA"). 

Considering claim 27, Ahmed discloses using token phrases like "we go now to" 
and "more on that" to indicate and identify a break or transition in a segment. It would 
have been obvious token phrases like "that is all" or "that's all" would have also 
indicated a break between segments since these phrases are often used in the news. 
Therefore, it would have been obvious to one having ordinary skill in the art at the time 
the invention was made to modify Ahmed to include the claimed phrases to quick 
identify breaks or transitions in the segment since these phrases are often used. 
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V. Claims 55 - 57 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Ahmed in view of Youman et al (5,629,733). 

Regarding claim 55, claim 55 recites some of the same limitations as recited 
above, and thus is rejected for at least those reasons provided above. Claim 55 further 
recites presenting a plurality of related story segments using a layered hierarchical 
presentation of the summary representations at a relatively high hierarchical level and 
the source data at a relatively low hierarchical level, together with hyperlinks permitting 
navigation among related story segments to a desired hierarchical level representation. 

Ahmed discloses displaying a plurality of related summary story segments (see 
stories for "Headline News" in fig 2-b) with source information (see "Headline News" 
icon 21 3 on the right) and hyperlinks 21 5 at the top (see fig 2-b). 

Youman teaches displaying hierarchical levels with a summary of a program 
located at the top of the screen with the source information at the bottom (see fig 24a). 
It would have been obvious from Youman placing the source information on a lower 
level of the screen would have provide a better viewing of the summary since the 
source information would not be in the way i.e. in the lower corner. Therefore, it would 
have been obvious to one having ordinary skill in the art at the time the invention was 
made to modify Ahmed to include the claimed hierarchical level to place the more 
important information like the summary information on top and the source of the 
information on the bottom to provide a more clearly prominent display of the important 
summary information. 

Considering claim 56, Ahmed discloses the claimed information is a text stream, 
the source data is text relating to a story segment and the summary representation 
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includes named entities (see fig 2-b, named entities met by "Erin: A Tropical Storm" the 
story segment is met by the summary description of "Erin: A Tropical Storm"). 

Considering claim 57, Ahmed discloses viewing image data (see fig 2-b) and 
thus discloses the claimed "information stream is an image stream". Further, Ahmed 
discloses displaying frames of information including a title and summary (see item 214 
fig 2-b, frame 214 meets "keyframe" limitation). 



Allowable Subject Matter 
VI. Claims 59-62 are allowed. 

Conclusion 

THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time 
policy as set forth in 37 CFR 1 .1 36(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1 .136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the mailing date of this final action. 
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VII. The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure. 

Janse et al (6,340,971) - key-frame based video display 
Blum et al (5,918,223) - segmentation of audio information 
Liou et al (6,278,446) - organization and browsing of video 
Yeo et al (5,821 ,945) - browsing based on content and structure 
Wissner (5,752,029) - editing multimedia composition 

Any response to this action should be mailed to: 

Commissioner of Patents and Trademarks 
Washington, D.C. 20231 
or faxed to: 

(703) 308-9051, (for formal communications intended for entry) 

Or: 

(703) 308- 5399 (for informal or draft communications, please label 

"PROPOSED" or "DRAFT") 
Hand-delivered responses should be brought to Crystal Park II, 2121 
Crystal Drive, Arlington. VA., Sixth Floor (Receptionist). 

Any inquiry concerning this communication or earlier communications from the 
examiner 

should be directed to Vivek Srivastava whose telephone number is (703) 305 - 4038. 
The 

examiner can normally be reached on Monday - Thursday from 8:00 am to 5:30 pm. 
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Art Unit: 261 1 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Andy Faile, can be reached at (703) 305 - 4380. 

Any inquiry of a general nature or relating to the status of this application or 
proceeding 

should be directed to the group receptionist whose telephone number is (703) 305 - 



3900. 




VS 6/1 3/03 



VIVEK SRIVASTAVA 
PATENT EXAMINER 



